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ABSTRACT 

Federal legislation mandates that applicants for 
research and development contracts in vocational education 
demonstrate a reasonable probability that the contract will result in 
improved teaching techniques or curriculum materials. lo facilitate 
the impact of research activities, one must develop procedures to 
predict and assess research inpact^ At present, the responsibility 
for impact assessment usually lies with researchers. This causes 
difficulty for some researchers who fail to see how impact assessment 
of their individual projects pertains to overall state program goals. 
Begirning this assessment process with clarification of funding 
program rather than individual project goals would eliminate this 
difficulty and foster inter- pre ject linkages. This approach helps 
eliminate the temptation to fund only low-risk^ low-payoff projects, 
helps establish mere realistic and attainable expectations, and 
encourages multiple-year efforts. Focusing on programs and program 
Qoals does net, however, eliminate the need for project designers to 
determine their own goals and methods for measuring impact. Intended 
immediate impact should be assessed during the funding life of the 
proiect, and long-range impact can be assessed as part of the 
evaluation of the success of the total program. (A proposed model 
impact statement is appended.) (MN) 
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^ I EDUCATION POSITION OR POLICY 

Wynette Barnard, Rupert N. Evans, flilai; lsnickerlj.K.!.ur, "permission to reproduce this 
ICS Earl Russc^n, and Carol SandcM-; material has been granted by 

Departnent of Vocational and Technical Eduction fc^^ipgrA A}. E.\far%s 

f-H Univc^rsity of Illinois 

|F— ^ TO THE EDUCATIONAL RESOURCES 

INFORMATION CENTER (ERIC)." 

These remarks are based largely on ^lie ro.Viit/, of coiiL inu ing proj'..:.L 
funded at the University of Illinois Ijv ifio Illinois iJ^^'i'd li; -w^L of Adull, 
Vocational, and Technical Education. The official l.itU; of t\\'i project 
is "Development of a System for Assessing the lifipact oK Voc-iLicnal 
Education Research and DeveloiJincnt on Vocational Educat:Ic>iu " Ou)- r>iiorL 
title is "Project Impact." 

(In addition to the many contri bn!-. ions oT [iroject" su'ifi, I also 
want to acknowled(j(.^ the contributions of Ron flcCage to the early evolutlcn 
and subsequent devc?lopment of the project. His posiLicui f^s director of 
program improvemenL in the lJe[)artment of Adult, Vocat.-iorn! cuid Technical 
Education has been filled ably by John l.'asiibui^n , v;ho continues to 

work closely with us, but fv^f^ is tlio -..crt oT i-c-rson v:ho never 1)0 
replaced. By acknov/ledgir.'j Ron at this poirrt, I an not trying to co-opt 
him as a discussant. He may very v/ell take issue with .,Oii:e of what I say 
at this symposium! ) 

Project Impact to Date 

By v/ay of background, you should have a brief review of our progress 
to date. "Project Impact" is iti its \hird year of funding. The purriosc 
of the project has been to identify and develc'p procecli,;i'.'':. Tor coiiiplying 
^ with the impact requirements of Public Law 94-482. To i.lri^- end, •'Proj:xi: 

^ Impact" has studied five problems: (I) definr.- i, (2) hov; 

to assess impact, (3) !:ow to show cause and (^rfi.'Ct rc'l f. i dnsS-.i p:; iJ0tv;er'n 

Ml 

Q R&D activities and changes in the vocational educaLicii tf.s-^.c hiriw- 



learning situation, (4) how to predict llic pi~(>L\ilri 1 i ly wT ii,!;icict, and 
(5) how to facilitate the imijact of I{ I \) activiLies. 

It is the last [Problem, "how to facilitate the iminc:!: of R ?< D 
■activities", to which we arc addressing riiost of our prcsid^.t efforts. 
However, we continue to refine our tentative answei^s to (he (.1:0 pr^obloiiis 
of "how to assess impact" and "how to predict the prob.'ti)i 1 i ty of im:aU.L". 
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Our thinking is that we ncocl Lo spcciry tlio result/; l.o he (.'xp'/cLcfl 
through research efforts before wo can attcmpL to assess iinpafjt. -Jo 
need to specify goals (outcomes to be expected) prior to conducting 
research and development efforts. 

At the present time the rcsponsibi 1 il.y tor impact ossossi;)onL almost 
always lies with the researcho 3. In preparing a proposal for vocational 
education R&D funding in Illir^ois, researchers are rec|urrcJ to include 
a statement of intended impact of their proposed activity t.liat includes: 
(1) target populations expected to be affected, (2) quantdtive efforts 
expected for each target population, (3) qualitative efforts expected 
for each target population, (4) the expected time frame for each effect, 
(5) how the intended impact will be achieved, and (6) iiiethods of 
measuring/evaluating the activity's impact(s) intended and unintended 
effects, and qualitative and quantitative effects. The researcher, if 
funded for the activity, is then expected to update the fundirrj agreement's 
statement of impact with each quarterly i)rogress report. This requi rei;ient 
poses difficulty for some reseaixhers who fail to see liow tlio imi^act 
assessment of their individual projects pertains to overall goals cf the 
state department's program. It also poses difficulty for reviewers wlio 
rate project proposals without having a [)ersonal knowlechje of pronrai:i 
goals or of how to assess potential impact. Researchers cilso find it 
difficult to: (7) predict when the effects of their project activities 
will peak, and (8) conceive how the long range inij^nct of tlieir work can 
best be assessed. 

In an earlier phase of our work, we concluderl that "if educational 
research is to have positive educational imjoact, it sliouir! he charac- 
terized by continuity . • • Almost every project should In* jiart of 0 
coherent program of research in which problems are ade<; jate'ly inves- 

gated before solutions are demonstrated. Such pt^ogi^'i!;;:. of research 
are more likely to have impact if tliere is continuity oT effort on the 
part of the researcliers; continuity of goals and fundinq ori the [VD^t of 
organizations which fund research; continuity of successful researcli 
'results reported to consumers; and continuity of evali.'ation" (Chieney- 
Stern and Evans, 1979, p. 10). 
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To respond to tlicsc cuticcrns of continuity, v/o drc? (i(:vrjlc)i)i[ui a 
system of assessing the impact of vocational education U 6 D to usc^d 
by all involved, starting with the fundcr. In our proposed systcMii, 
each individual project is considered to be part of a larger program as 
defined by the funder (in our case, this is the R ?^ D section of tfio 
Illinois Department of Adult, Vocational, and Technical i^ducation). 

Earlier in our research we were involved in the "top-dov.'n" and 
"bottom-up" tracking of the outcomes of groups of projects wliich wo 
called "programs". For example, we looked at a group of [irojects which 
were all concerned with instruction in vocational agriculture. Later, 
we came to the conclusion that our groupings were somewhat artificial. 
Some of the project directors saw tlioir own projects as part of a 
personal research program, but generally, neither the project directors 
nor the staff of tlie funding agency saw the grouping of projects into 
"programs" in the same way that we did. Since then, we have come to 
believe that program goals and program components for tho research and 
development activities supported by each funding agency sliowld be defined 
by that agency. Each program is defined as a group of \i & D program 
improvement projects which is seen by the funder as having co::i;non or 
related goals. The intended program impact (long range) is based, 
therefore, on the degree of achievement of similar goals by individual 
projects. It is unlikely that these goals could be acliieved totally by 
any one project. Short term (interim) intended impact is based on the 
achievement of individual project goals during the duration of the funded 
efforts. In addition to intended impacts, all of win'cli are likely to be 
positive, there are unintended impacts, which may be either pnsitive or 
negative. Outcomes of programs are tlie algebraic sum o! Ih'.* cont)^ibutions 
(intended and unintended) made by all of the individucil (irrijccts within 
that program. The sum of the contributions of each of the !i ?. D agency's 
programs constitutes tlie impact of that agency. 

All of this is easier to say tlian to do. [lo expr^:!. to continue 
searching for knowledge about' impact for some Lime to cowx'. [n tlie next 
year, "Project Impact" activities will be designed to as^.ist funders, 
researchers, and practitioners to understand thoroughly Llie compo;!out5; of 
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tho impact assossr.ient syst(?!ii. Individual rolos and nr^ijun'^ 1 bi I i t ios i. i I 1 
bo clarified. To assist researchers with tlioir task of I'v.scs^, iiiv; '.hui l 
term (interim) impact, we are developing a handbook to facilitaU; the? 
assessment of impact that occurs during the lifetime of j^rojecLs. 

The Need for Speci fy1ng f:Xjl£c:.tGd Im[3act 

Four years have passed since COVERD reported that Liiciy could not 
find documented impact, of the results of millions of dollar.i of funded . 
vocational education research arid devel opincnt acLiviLics. Concirov.s 
used this lack of documented impacl. as the justification for iiiandati n;) 
consideration of impact during the funding process. PL rc-ciuiroi 
that applicants for k & D contracts in vocational education "d(.T::ons Lratc 
a reasonable probal)il ity" that the cotitract will result in improved 
teaching techniques or curriculum materials tliat would he used in a 
"substantial numl^or of classrooms or other learning situations within 
five years after termination of such contracts" (94tfi Cc;- qross, 1976), 
Since the imposition of that mandate, not only have additional mi 11 ions 
of dollars been used to fund research in vocational education, !vjt a 
substantial amount of money has also gone into rcsoarchiru; how we can 
comply with PL 94-482 and how we can measure the i'iiiidCL oT research 
and development. Researcliers may quarrel with the lan']u'^''^g'^? of the law 
and with its emphasis on immediate payoff for eacfi and e\\?ry research 
project, but few would argue against tiie need for dcMiionsLraLing the 
impact of research programs, whicli seems to us to be tra- spirit of 
the law. 

Starting With Programs 

Perhaps v/e have spent too much time and energy try in^j to ficjiire 
out how to measure project results or outcomes relative.* to cn'ovl 
we spend on specifying program goals and ol)jcctives. liio iiorci clearly 
and consistently we can specify goals and objectives, L!;e cosier tlic 
job of specifying expected results. 

Often project directors do not see a clear linkagr.- hfjLwoi^n their 
•individual projects and the overall plan of tlie funding aguncy, e.g. tlie 
state RCU. Individiial project goals should relate to P.CU prugram goals 



and in turn these program go.'.ilr. slicnild )*(?laLo Lo tir.* (»vt'rMll (jool'. (jf 
the RCU. The identification of tlris linkage is ouo arc.:i v.iiicli often 
has been overlooked or at least riOt clearly stated to individual project 
directors. Project directors often do not know that programs exist, 
let alone know the |.)rograin goals to wliicli thoy should rijlatc*. iJofining 
programs and program goals shou'.d be one of tlie first sLuj-s in R D 
planning. 

Ideally each funding agency (e.g. RCU) would identiry and dc-Tino 
what constitutes its programs. Preferably this sfiould l.:"^ done during 
the priority development phase, as describeci by Ron McC^/jt^ (n^'O). 
During this priority development pliase, state RCU's gen;_rally gather 
input from conferences, previous research reports, educational 
institutions, and surveys of constituent grou[)S in order Lo establish 
priorities for tho following year. These prioiMties could then bo 
organized into programs which would hel[i conct^ptualizc an !: & D [ilan 
for the RCU for a specified period of tiiiie. Lxamplos oT \r'.\c'u programs 
might be the service areas of agriculture, business, etc. as well as 
general programs such as S['Gcial nc-^e:::^, worl:-cd'jcat1ofi , adn-inistratioii, 
and interdisciplinary curriculum. Or they iniglit be toj-ics such as 
productivity, access, or job satisfaction. 

Defining programs will assist project directors in id^iiLifying 
with the program on which they need to focus tlu'^ouglioii t tiir.-ii* project 
efforts. It will also identify other projects which liavo or arc 

seeking to achieve tlie same program goals, tlius fostering intor-projccL 
linkages. Defining programs in this way will help to CJ^ t aijl i sli the 
connection between the research process and its expecU.'! iiii'cict. 
Contributions to the well being of target populations will bjcoii^j 
clearer. 

By identifying its own programs, each runding ag=.M»":y (^'.g. I^f-iJ) 
maintains control of the type and numl)e)* of prorjra:,is \vr wliich it is 
responsible. Eacli program probably sliould liavc no l(-.s tlvni !.v:a <ind 
no more than ten related projects at any one time. [^'Klt slal"e will 
'have a different number of programs and projects. Th::' fiumbor or 
programs it funds should be based on assessed needs afv! liioney aV()ila!)lo. 
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The number of projects and their size should bo based on ir.o nouf;:. of 
each program and tlie speed witli which those needs should be n.-L. 

Moving t£ Goals and Objectives 

Establishing program goals is the next step which i^iust be u!uIortiii:oii 
by the funding agency (e.g. RCU). Determining program cio^ils not only .rid 
the process of planning for R & D but also identifies tho c'jtcon;:r; to be 
expected as a result of research rein ted to the prograii!. Progrdu goalr. 
such as curriculum improvement would necessarily bo broodLT thar. project 
goals. The expectation is that several individual projects will be 
designed to achieve different aspocis of the same major prograiii yoal . 
This system provides a means by which related individual projects can 
tie their goals to eacli other and to the goals determined by Mie RCU. 

Projec t Goals 

Once program n^^cils have been established, specific obji\:Lives for 
meeting the goals can be determined. It is at this point that the 
linkage between iridividuai projects and vocational educ^l.ion 11 & D 
programs can be clarified. It is lil.ely i^-l prograni objectives will 
serve as a basis for guiding the development of project gonls. in the 
process of determining project goals, a project director can look at 
the "overall picture" (represented by program goals) and ad^ipt those 
to local needs and goals; thus designing a unique set of rioals for 
an individual project. Conversely, the funding agency will seek to be 
sure that the sum of individual project goals will meet program goals. 

"The Lawn Party: The Evolution of Federal Progr^::./. in Local 
Settings," (Farrar, E. , et al , 1980) describes how the LrrMiol aL i on of 
federal programs to local settings can be compared to Liv;.'i p(:)-ty. 
The federal program is tlie occasion for the gathering, '-nd all of the 
guests (local administrators, teachers, parents, bo(}rd \'j:u\h- :]•'■.) iiave 
come for different reasons, because they have different, concerns at 
home. Thus, they each liope to adapt one federal progra!ii in different 
ways depending on their local goals and needs, which in turn lead:, to 
•a broad range of possible outcomes. This process may l;e similai' to the 
development of iridividual project goals. Al th.^ugh i)rr)r:rriiii goals (U^id 
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objectives will help to guide tlie foniiiilaLion of |)roj(u:L ^joalG, a nerjoLi uioii 
process must take place aniony Furulers, rosoarchors ar)(' ptMci. i (.ioncrs to en- 
sure that project goals meet local needs, v/hile sLil l mn inLa iniiKj tho l ink- 
age between programs and pt^ojccts. Tlie outcomes dcsirod l^y Uio funding 
agency, the outcomes desired by the agcMicy conducLing Im-.: projcr.l:, atid l.iiu 
outcomes desired by LEA practitioners ore orion suinuv.ii.:;!: rlifrcrent. 

One might think of tho goals of research as beincj Lhc: !iul) of a v/hool, 
with the spokes being programs arul program goals. A ni:.i!').:r oT projects 
v/ould reinforce each spo!;e and in turn each project v/ouici bc^ su;)por'tod hy 
a number of local interpretations of the outputs of a projcjct (sec 
Program and Project Relationship model below). 
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Assessing Impact 

Using program goals as a basis for spnciPyiiKj ^-erji'l l:> Lo -x: obuciiiK.'cl 
through research also provides a means for assess incj the iiniKicL of 
vocational education R & D, in the spirit of the manddtod V\. 94-';32. 
Moore and Magisos (1977) supported this idea after revit:winn tlice COVrRi) 
report. They concluded that "total evaluation oiicjht Lo b-: procjroinniatic, 
linking several projects together, so Lhat summati ve cva luaUon is 
broadly based." 

We define impact in our field as "measurable or olirajrvabl cliannes/ 
influences, positive or negative, that result froi;i a vocat.iuiinl 
education R&D program improvement activity." It is possil)lo to assess 
the impact of related individual projects, whicli conihinc i.o rorii: a pro- 
gram, and to compare this assessment vnth the prograDi cjools v.'hicli were 
intended. 

There are several advantages of v/orking within the fr^si y./or!: of 
programs, rather than projects: 

1. It helps to eliminate the temptation by RCU's lo 
fund only low payoff, low risk projects bociiuso 
they have a high likelihood of shov/iiuj a por.itivu 
impact, in order to fulfill PL 94-482 iiiipact 
requirements. 

2. Individual project directors can see tlioir projoi.t 
efforts as contributing to an overall j)laii micl sd. 
of goals, thus tying their project goals and irn icl: 
assessment to a larger whole. 

3. It may aid project directors in setting mor^' 
realistic and attainable expectations. 

4. It encourages multiple-year efforts, thus rclicvihj 
the unrealistic urge to solve |)ersisLont [jrcib' 

in a single year. 

Although high risk, high payoff projects may not show iii.pact within 
five years they have the potential to contribute a great do.il to resccirr.h 
knowledge. Assessing the impact of programs allows for fundi nci soiiuf 
individual projects which may fail, without seriously alfc^ctiiuj the 
overall attainment of program goals. 
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Focusing on programs and program goals docj not c i iMinato tho 
need for designers of individual projects to deLeniiinc Uieir own goals 
and methods of measuring impact. Each project needs to be re:.ponsible 
for identifying outcomes and assessing impact throughyjL Uic> Tunded 
life of the project. Individual project goals and expoctud y^GF.ults 
(outcomes) will vary according to the: (1) pronraiu of whicti it is 
a part, (2) phase of the project (research, developnieru dissen^'n- 
ation), (3) target audience, (4) proposed R D outputs, (5) dis- 
semination strategies, (6) intended effect upon the targot audience, 
(7) extent of intended effect, and (8) method of proposed measure- 
ment. Addressing these considerations during the [)roposal dovelopinont 
phase of a research project will assist proposal writer:, with tho. 
task of developing th^ir required statements of intondoc; impact 
(see Appendix A for samples of impact stateinents) . 

The intended immediate impact of a project relatL-s to project 
goals and much of this impact could be assessed during the fiificled 
life of a project. The intended long range impact relates tcj 
program goals and can be assessed as [iart of the evaluation of the 
success of the total program. Time lapses between procjrai:! [);iasos 
can be utilized to measure changes in participant behrjvior and the 
effects of changes on their organizations. In additiof^, the moasui^o- 
ment of effects resulting from the first phase of a fvro.jrctiii can be 
conducted during succeeding program phases (Knox, 19/9). 

We believe this approach to specifying the extend:.! r^-suits of 
research in vocational education has potential for d ■'; -i ing the 
impact of R & D. By concentrating on programs, ratliT :»rojerts, 
we believe that the funding agency, those who condncl: i'yr.jecLs, and 
those who are the target audiences of project efforl!. riro nioro likely 
to see positive R&D impact. 

Vie recognize that documenting the results of r- •>L'aroh h] 
. vocational education is costly, particularly the doouKiontation of 



io 



10 



ultimate impact. Appendix A is a tentative liioclel sLiit.ci.^il of iiiij^act. 
to be included in a project proposal. Although it tias bc-un j^arcd down 
from more grandiosf.:: models, the budget required would sii i l in tlie 
thousands of dollars, and this estimate does noL include.' Lli:: Lii:i(} tlioL 
the state staff and LEA staff would spend during tl)c an^c:''.s^llclll. 

We are really concerned with three aspects of iiiipdcL. Tlic first 
is to increase awareness of the concept oT R D iiirpjct .mci tc^ conviiiC'- 
proposal writers, proposal reviewers and project nicMiitor-, lU-'l iinijact 
and the assessment of impact are needed. Tfiis firr>t slty is 
relatively inexpensive. 

The second and third steps are tlie actual productioit of inij/act 
and the documentation of this impact. Borli of these can tc- cjinte 
expensive for some projects. There seems to be litl:le Sv-nst.' in 
spending $5^000 on the impact of a $10,000 project. It 1--. essential, 
however, that each program (not necessarily eacli projoc': ) f;:ivo 
impact, and have documentcition of this impact. The cf'V.ts of tl^oso 
activities must be borne somehcv;, eve:i if it j^iiisults in reJiiction 
of some research production. One of tlie major tasks foccd by our 
project is how to reduce as far as possible the costs of iii.pact anc! 
impact assessment while increasing awareness of tl. . rier.;.| io)^ ii:i:"iact, 
increasing the amount of impact, and increasing tl]o docJiriontation 
of impact. 
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PROPOSED MODEL IMPACT STATF.MEMT 

Research Phase 

The Project vn'll di ssemiricite the final report to tl;e mcinbors of 
the State Vocational Education R&D staff. Within tv.'O luonLhs after 
delivery, a meeting of tlie State and Project staffs will be held to 
discuss the report and excliange ideas on the findings. The iniiiiediato 
impact of the Project will be to create an awareness of tho state of 
the art (of this topic) by 100/;; of the State staff. Tfiis impact will 
be measured by attendance at the meeting and through oh^nrvation of 
the discussion. The ultimate impact will be the use of tl)ic report by 
the State staff as a foundation for developing guidcolinor. for a cur- 
riculum development project. Tliis impact will be aclriovod tlirougli 
the State planning meetings and can be assessed by evidence of an 
RFP for a curriculum development project based on this research, 
within tv/o years. 

Development Phase 

It is expected than an immediate impact of this Project will be 
an increase in communication between home ecoiiomics teachL^rf. in the 5 
field test schools and the Project staff, resulting in curriculum 
materials which are relevant to high school sophomore students' needs. 
This will be achieved by monthly meetings between thti brojoct Director 
and teachers who are field testing the materials. Ass^jssiijnt cf tliis 
impact will be made by an opinionaire sent to tlie teticli-Ts at the end 
of the field test period. It is expected that of responding 
teachers will indicate tliat there was ample opportunity Cor tlioni to 
provide input into the revisions and tliat an open coi;i:::uiricat io]i network 
had been achieved. 

The curriculum materials will bo disseminated to at least 150 h1gii 
school home economics teachers through four regional worksliops con- 
ducted by the Project Director. It is expected that IvO,; of tlie 
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participants will adopt the curriculuni inaLerials within one year of 
the workshop. This will be assessed by a follow-up mV\ survey. The 
ultimate impact will be a significant increase in the eni'onnient of 
the home economics child case program by students who iiavc used the 
new curriculum materials. This assessment will be mode by comparing 
results of a three year follow-up study with the baseline data 
collected in this study. 

Di ssemination Phase 

The curriculum materials on food safety will be di ssominatod to 
100 or more practicing teachers of restaurant management at the 
secondary school level. Regional workshops will be held by the 
project director to provide an opportunity to discuss how the material 
can be used in or adapted to individual classroom si tualicMis . It is 
expected that 707 of the tenchors in nttendance will try th'j i:interials 
and adopt them foi* use in Lrio classrooni the semester fo 11 cjwing dis- 
semination. This will be assessed by random sai.iph^ Lelophone 
survey at the end of the Si:r.'.:st,?r. The ii^-r-ct of the curriculum 
materials on high school seniors enrolled in restaurant i:i:ifV'^.n'jmont 
will be an increase in knowledge of food safety [jrincipl^*s. This 
impact will be assessed within one year hy comparing tir^ pei-centage 
of students who used the new curriculum matei^ials and wiic; [>assed the 
state licensing exam on food safety to the overall pcrc'MitcKje of 
secondary students who passed the exam. It is expected that there 
will be a significant diffeiTjnce between the two grouj)^. in favor of 
those who used the new curriculum matei^ials. 

The ultimate impact will l)cj that stuflents will np: l.v 'his 
knowledge of food safety on the job by pr^acticing sa [ -j i"ci:)d han Jl iivi. 
This impact could be assessed ttn^ee years after dissi'inin-iL ior; hy 
conducting case studies of five studeiri/-. "who used ttr^ ikjw cm-r iculuin. 
The case studies would include ()n-th(^-j()l} ulr:erv.it ion and !!i-(ii;jiLI; 
interviews with the studerit and em|iloyer to rletermino c v i cii;;;',:e of 
application. Funding for this assessment is not )'e:rics1(^d now, and 
conduct of it will lie contingent on the needs of the? st il.c fur 
documentation of ultimate impact. 



